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(57) Abstract 

A digital image warper system (5) produces a warped output image from an input image. The warper system increases the sampling 
rate of a sampled image above Nyquist rate or samples an analog image at a higher rate than the Nyquist rate and prefiltcrs the upsampled 
image to counteract filtering which may occur during the warping operation (10). The upsampled image is warped using an interpolator 
for resampling to produce a warped upsampled image (20). The interpolator is. for example, a low quality interpolator such as a bilinear 
interpolator. The warped image is ttten down-sampled to the same resolution as the input image to produce the warped image (30). Down- 
sampling and warping can be combined into one step by modifying the geometric transformation function implemented when warping the 
upsampled image. 
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COMPUTATIONALLY EFFICIENT DIGITAL IMAGE WARPING 

Fie^ld of the Invention 

This invention relates to a method and apparatus of 
digitally processing image data and, particularly, a method and 
apparatus for performing a high quality digital image warp. 

B^qkqx-cpuna gf the Invent ign 

There are two basic components to an image warp: (1) 
spatial transformation and (2) resampling through 
interpolation. A spatial transformation defines a geometric 
relationship between each point in an input and a warped image. 

Inverse mapping maps a pixel value in the warped 
image to a pixel value in the input image which may be located 
between pixels in the input image. Therefore, it is desirable 
to use interpolation to generate the pixel value from the non- 
integral positions in the input image. 

Current hardware approaches for performing the 
interpolation required for resampling use a two-by-two 
neighborhood of pixels around the address value of the pixel in 
the source image to calculate each pixel value in the warped 
image. This is commonly called bilinear interpolation. 
Bilinear interpolation uses a local neighborhood of four pixels 
and a weighted average of those four pixels for the 
interpolation. For a real-time implementation four pixel 
values are accessed simultaneously every clock cycle. The four 
pixel values are then multiplied by the appropriate weights and 
summed to produce a pixel value in the warped image . 

One such system for performing bilinear interpolation 
is described in Real-time Bilinear Interpolation Using the 
TMC2 3 01 by Steve Gomez, TRW LSI Products Division, and dated 
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January 21, 1989. This system uses four separate memories that 
can be accessed in parallel when performing bilinear 
interpolation. The system also includes a look-up table for 
storing the coefficients to determine the weighted average. 
5 The weighting coefficients are multiplied by respective pixel 
values from the input image and, then, summed to produce a 
pixel value in the warped image. Real-time Bilinear 
Tn^^rnolat ■i^r. Ticking the TMC2301 is herein incorporated by 
reference for its teachings on bilinear interpolation. 

10 The next higher quality interpolator uses a three-by- 

three pixel area in the input image to compute each pixel value 
in the warped image. The complexity and expense of this type 
warper dramatically increases relative to that of a bilinear 
interpolator. Nine separate memories, nine coefficients, and a 

15 nine term sum of products is required. In applications where 

size, power, and cost are at a premium, this is an unacceptable 
solution. If a better interpolation is required, for example, 
four-by-four or greater, this problem is compounded. 

To overcome these shortcomings, a new warper method 
20 and system is provided. 

.qummarv of the Invention 

The present invention is directed to a digital image 
warper system which produces a warped image from an input 
image. The warper system increases the sampling rate of a 

25 sampled image above the Nyquist rate or samples an analog image 
at a higher rate than the Nyquist rate. The upsampled image is 
warped using an interpolator for resampling to produce a warped 
upsampled image. The interpolator is, for example, a low. 
quality interpolator such as a bilinear interpolator. The 

30 warped image is down sampled to the same resolution as the 
input image to produce the warped image. Down sampling and 
warping can be combined into one step by modifying the 
geometric transformation function implemented when warping the 
upsampled image. 
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It is to be understood that both the foregoing 
general description and the following detailed description are 
exemplary, but are not restrictive, of the invention. 

Brief Description of the Drawing 

5 The invention is best understood from the following 

detailed description when read in connection with the 
accompanying drawings, in which: 

Fig. 1 is a block diagram of an exemplary embodiment 
of the present invention directed to a digital image warping 
10 system. 

Fig. 2 is a block diagram of upsampler 10 shown in 
Fig. 1 according to an exemplary embodiment of the present 
invention. 

Fig. 3 is a more detailed block diagram of upsampler 
15 10 shown in Fig . 2 . 

Fig. 4 is a data structure diagram illustrating the 
conceptual operation of upsampler 10 shown in Figures 2 and 3. 

Fig. 4a is a representation of the upsampled image 
signal 2 07 where the pixel values in the upsampled image signal 
20 207 have been renumbered. 

Figs. 5 (a) -5(c) are respective Fourier transforms 
which illustrate exemplary spatial frequency spectra of the 
input image signal 205, intermediate image signal 206, and 
upsampled image signal 207 shown in Fig. 4. 

25 Fig. 6 is a block diagram of warper 20 shown in Fig. 

1 according to an exemplary embodiment of the present 
invention . 

Fig. 7 is a data structure diagram illustrating the 
input image, the upsampled image, and the warped image. 



3 



BNSOOCIO: <WO 97394aOA1J_> 



wo 97/3M20 



PCT/US97y05427 



Figs. 8 is a flow chart diagram illustrating the 
operation of warper 2 0 shown in Fig. 1 in accordance with an 
exemplary embodiment of the present invention. 

Fig. 9 (a) shows respective Fourier transforms of the 
upsampled image signal 207, the intermediate warped image, and 
the warped image 700 illustrating the operation of the down 
sampler 3 0 in accordance with an exemplary embodiment of the 
present invention . 

Fig. 9(b) is a Fourier transform of a input image 
signal using a bilinear interpolation which does not include 
upsampling of the input image signal. 

Figs. 9{c) - 9(f) are enlarged views of the data 
structure diagram shown in Figure 7 . 

Fig. 10 is a block diagram of another exemplary 
embodiment of the present invention directed to a digital image 
warper implemented on a single integrated circuit (chip) . 

Figure 11 is a timing diagram useful for explaining 
the operation of the warper chip of Fig. 10. 

n^rail^d Description of the Tnvent;j,(?n 

Overview 

The exemplary embodiment of the present invention is 
directed to a digital image warper system which produces a 
warped image from an input image. The warper system increases 
the sampling rate of a sampled image above the Nyquist rate. 
Typically, digital images to be warped are sampled ac the 
Nyquist rate. Increasing the sampling rate reduces the spatial 
frequencies represented by a fixed number of samples. The 
increased sampling rate is accomplished by sampling the analog 
image at a higher rate than the Nyquist rate or by digitally 
processing an already ISFyquist sampled image to increase its 
sampling rate (e.g. by doubling or tripling each sample) . The 
later of these two approaches maintains the modularity of the 
warper system within a larger image processing system. 

4 
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The upsampled image is warped, using an interpolator 
for resampling, to produce a warped upsampled image. The 
interpolator can be a relatively low quality interpolator such 
as a bilinear interpolator. The warped image is then down 
5 sampled to the same resolution as the input image to produce 
the warped image. Down sampling and warping can be combined 
into one component by modifying the geometric transformation 
function implemented when warping the upsampled image, 

DESCRIPTION OF EXEMPLARY EMBODIMENTS 

10 Referring now to the drawings, wherein like reference 

numerals refer to like elements throughout. Fig. 1 illustrates 
the digital image warping system 5 according to an exemplary 
embodiment of the present invention. An input image signal 205 
is provided to upsampler 10. The input image signal 205 is 

15 sampled at or above the Nyquist rate and is a complete 

representation of the underlying analog image. Upsampler 10 
interpolates the input image signal 205 by a factor of, for 
example, 2", where N is a positive integer greater than zero, to 
increase the number of samples, i.e. pixel values, in the input 

20 image signal 205. Interpolation is not limited to a factor of 
2\ 

Upsampling is performed in the vertical and 
horizontal directions of the input image signal 205 using the 
same sample multiplication factor. For example, if the input 

25 image 205 has 512 pixel values by 512 pixel values, upsampling 
by a factor of two produces an upsampled image signal 2 07 
having 1024 pixel values by 1024 pixel values. Upsampling by a 
factor of four produces an upsampled image signal 207 having a 
size of 2048 pixel values by 2048 pixel values. The input 

30 image signal 205 is, for example, 256 pixel values by 2 56 pixel 
values. Thus, upsampling by a factor of two produces an 
upsampled image signal 207 having 512 pixel values by 512 pixel 
values . 

Alternatively, upsampler 10 can be provided an analog 
35 input signal. In this configuration, upsampler 10 samples the 

5 
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analog input signal at a rate of 2" times the Nyquist rate to 
produce the upsampled image signal 207. 

Upsampler 10 is a high quality interpolation filter 
shown in Figure 2 . Upsampler 10 upsamples the input image 
5 signal 205 by a factor of two in both directions using a 

separable filter, Upsampler 10 slightly peaks the frequency 
components that will be attenuated by warper 20, shown in 
Figure 1, to improve the image quality of the warped image. 

The input image signal 205 is provided to vertical 
10 filter 210 and vertical filter 220 directly and via lines 

delays 200 each of which delays an entire line of the input 
image for one line period. Vertical filter 210 is a unity 
filter and passes the pixel values of the even lines of the 
input image signal 205. Vertical filter 220 is a four tap 
15 filter with the coefficient values selected to be a sum of the 
powers of 2. The coefficient values are -.09375, .59375, 
.59375, and -.09375 which are respectively -3/32, 19/32, 19/32, 
and -3/32. The multiplication operations used to implement 
vertical filter 220 are simplified to arithmetic shift and sum 
20 operations by using these coefficient values. 

The pixel values from vertical filters 210 and 220 
are provided to (1) horizontal filters 230 which are unity 
filters and pass the even pixel values and (2) horizontal 
filters 240 which are four tap filters. The coefficient values 
25 of horizontal filters 240 are selected to be a sum of the 
powers of 2 and are -.09375, .59375, .59375, and -.09375. 

Figure 3 is a more detailed block diagram of 
upsampler 10 shown in figure 2. Pixel values from a line of 
the input image signal 205 are provided to input terminal IN. 
The lines of the input image signal 205 are processed through 
line delays 200, adders 300a-300f, and multipliers 310a^310e to 
vertically interpolate the odd lines of the input image and 
produce vertically interpolated lines Vodd which become the odd 
numbered lines of the upsampled image 207, shown in Fig. 1- 
35 Coefficients CO, CI, and C2 are 1/16, 1/32, and 1/2, 

respectively. The output signal Veven of the upsampler 10 are 

6 
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not vertically filtered • These lines become the even nximbered 
lines of the upsampled image 207 and represent the image 
samples of the input image as they were received. 

The even lines Veven are provided to and processed 
5 using pixel delays 320a-320d, adders 300h-3001, and multipliers 
310f-310i to horizontally interpolate the pixels in the even 
lines Veven to produce output pixel signal OUT2 which 
represents the odd pixels in the even lines of the upsampled 
image 207, shown in Fig. 1. The even pixels in the output 

10 image signal are not horizontally interpolated and are provided 
as output pixel signal OUTl, The vertically interpolated lines 
Vodd are provided to and processed using pixel delays 320e- 
320g, adders 300m-30Cr, and multipliers 310j-310n to produce 
horizontally interpolated odd pixels in the vertically 

15 interpolated lines Vodd to produce output pixel signal 0UT4 . 

The pixels of the vertically interpolated lines Vodd are also 
provided, without further interpolation, as pixel signal OUT3 . 

Figure 4 illustrates the conceptual operation of 
upsampler 10 shown in Figures 2 and 3. Figure 4 is a spatial 

20 domain representation of the sampled images which illustrates 
upsampling by a factor of two (N = 2). Zero data values are 
•inserted into the input image signal 205, as shown in Fig. 4, 
to produce an intermediate image signal 206. A, B, C, and D 
represent pixel values in the input image signal 205. 

25 Intermediate image signal 206 is low-pass filtered to produce 
upsampled image signal 2 07 having pixel values A, B, C, and D 
plus interpolation pixel values I. By implementing this 
conceptual process, the Nyquist sampled image 205 is critically 
sampled. 

30 Inserting zeros into the input image signal 205 

compresses the Fourier transform of the image by a factor of 
two as is shown in Figures 5 (a) -5(c). Figures 5 (a) -5(c) are 
respective Fourier transforms of the input image signal 205, 
intermediate image signal 206, and upsampled image signal 207. 

35 A comparison of Figures 5(a) and 5(c) demonstrates that region 
400, shown in Fig. 5(a), over which the Fourier transform of 
the input image signal 205 extends is reduced in size to region 

7 
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401 in the upscunpled image signal 207, shown in Figure 5(c). 
Further, as is shown in Figure 5(c), a single copy of the 
compressed transform centered around every integer multiple of 
2n remains after the intermediate image 206 is low pass 
filtered. 

The upsampling kernel is a low-pass filter, with a 
cutoff frequency determined by the extent to which the input 
image signal 205 is upsampled. The low-pass filter eliminates 
frequencies except those compressed copies of the original 
signal that are centered on integer multiples of 2k. If the 
frequency characteristics of the interpolation method employed 
in the warper 20, shown in Fig. 1, are known, a better 
upsampling kernel can be selected to provide an overall warper 
system 5 with less image degradation. 

The upsampling kernel is selected to compensate for 
the deviations from a perfect low-pass reconstruction filter 
that may exist in the warper 20. For example, if the warping 
interpolation method is bilinear, described below, after 
upsampling, there will be a slight attenuation of the high 
spatial frequencies. The upsampling kernel of upsampler 10 is 
chosen to slightly peak these same frequencies to minimize the 
overall warper system 5 effect on the high spatial frequencies. 
Thus, if the warp interpolation method is known, an upsampling 
kernel for upsampler 10 can be selected to compensate for these 
deviations from a perfect low-pass filter by pre -amplifying or 
pre-attenuating frequencies in the input image signal 205. In 
this way, the overall degradation of the system can be reduced. 

Different warp transforms attenuate different high 
spatial frequencies when bilinear interpolation is used. For 
example, a half-pixel translation causes the most attenuation 
of the high spatial frequencies while integer translations have 
no attenuation. A single upsampling kernel between these to 
extremes can be selected to improve the overall image quality 
for different transforms, 

8 
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The upsampler generates pixel values in the upsampled 
image signal 207 at four times the rate pixels in the input 
image are provided to upsampler 10. Pixel values from the 
input image signal 2 05 are provided to upsampler 10, shown in 
5 Figure 1, at a rate of one pixel per clock cycle and supplied 
from upsampler 10 at four pixels per clock cycle. The four 
pixels are provided to warper 20, shown in Fig. 1, and stored 
in, for example, separate memories 600a- 600d, shown in Fig. 6, 
as explained below. 

10 Warper 20, shown in Fig. 1, warps the upsampled image 

signal 207, shown in Figure 4. Warper 20 is described with 
reference to Fig. 6. In Fig, 6, the pixel values of the 
upsampled image signal 207 are stored in memories 600a-600d so 
that each one of the memories stores one fourth of the 

15 upsampled image 207. The pixels in the warped image are 
interpolated from the stored pixel values using bilinear 
interpolation. Because the warp memory 500 is implemented as 
four separate memories 600a- 600d, for use by the bilinear 
interpolation, the upsampled image signal 2 07, shown in Fig. 4, 

20 can be stored in the warp memory 600 at four pixels per clock 
cycle. In this way, no backlog delay is incurred in the 
upsampler 10, shown in Fig. 1. 

Bilinear interpolation uses a local neighborhood of 
four pixels and a weighted average of those four pixels for the 

2b interpolation. The neighborhood of pixel values are the 

nearest pixel values to the source pixel value. The source 
pixel value 72 0 and the four closest neighbor pixel values 
720a-720d in the upsampled image are shown in Fig. 7. Input 
image 2 05 and upsampled image 2 07 have a pixel value at each 

30 vertex formed by the horizontal and vertical lines shown in Fig 
7. Pixel value 700 is a pixel value in warped image 200 which 
corresponds to source pixel 720. Pixel 700 also corresponds to 
pixel 710 in the input image. 

In a real-time implementation, four pixel values 
35 (720a, 720b, 720c and 720d) stored in the warp memory are 

retrieved at substantially the same time every clock cycle for 
use by the bilinear interpolator. These pixel values are then 

9 
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multiplied by the appropriate weights and summed to produce a 
pixel value in the intermediate warper image. The pixel values 
of the upsampled image signal 207 are stored in memories 600a- 
600d so that every grouping of the nearest pixel values 720a- 
5 720d to the source, pixel value may be accessed concurrently. 

Write address generator 615, shown in Fig. 6, 
produces addresses for storing the pixel values in the 
upsampled image signal 207 in memories 600a-600d. Upsampler 
10, shown in Fig, 1, produces a two-by-two array of pixel 

10 values [(A,,, E^^, F, G,,,, ) , (B,,, E, , , F, G,^,), (C,,,, E, , , F,,, 
G^^,,). or (D,^, E, ,, F, ,, G,.,, ) ] for each pixel value in the input 
image signal 2 05, shown in Fig. 7, where k and 1 are integers. 
The four pixels in the two-by-two array are stored in separate 
memories 600a-600d at the same address in each memory 600a-600d 

15 and at substantially the same time. In the exemplary 

embodiment of the invention this is implemented by storing each 
of the output samples, OUTl , OUT2 , OUT3 and OUT4 of the 
upsampler 10, shown in Fig. 3 in a respective memory 600a, 
600b, 600c, and 600d, 

20 For example, assume the input image has 2 56 pixel 

values by 2 56 pixel values and the input is upsampled by a 
factor of two. Then the upsampled image 2 07, shown in Fig. 7, 
has 512 pixel values by 512 pixel values where each square 
grouping of pixels 210, shown in Fig. 4, has a pixel located in 

25 a separate memory. The pixels values shown in Fig. 4 are 

stored in memories 600a-600d. Fig. 4a is a representation of 
the upsampled image signal 207 where the pixel values in the 
upsampled image signal 207, shown in Fig. 4, have been 
renumbered as Zl through Z4 for describing where the pixel 

30 values in the upsampled image are stored in memories 600a-600d. 

Each memory address x^ and y^ is twelve bits if the 
maximum size is 2^' X 2'' {4096 X 4096). Each memory address x, 
and y^ includes a least significant address bit x, and y„ which 
together specify the memory 600a-600d in which the pixel value 
35 is stored and location address bits x,,., and y,, , which together 
specify the address at which the pixel values are stored. The 

10 
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mapping of address bits and y^, into memories 600a- 600d is 
shown in Table 1 below. 



TABLE 1 



^0 


y. 


MEMORY 


0 


0 


600a 


0 


1 


600b 


1 


0 


600c 


1 


1 


600d 



Fig. 4a and Table 2 below illustrate the arrangemenc 
of the pixel values of the upsampled image 207 in memories 



600a- 


600d addresses x[ll-l] 


and y[ll 


-1] . 










TABLE 2 






xCll- 


1] y[ll-l] 


MEMORY 
600A 


MEMORY 
600B 


MEMORY 
600C 


MEMORY 
600D 


0 
0 
0 


0 
1 
2 






Z3o.o 
Z3„.. 
Z3o., 


Z4o.., 
Z4,, 
Z4„., 


0 

1 
1 


255 

0 

1 


21o.250 

Zl,.o 
Zli., 


Z2,o 
Z2,, 


Z3,.o 
Z3,, 


Z4„.35, 

Z4,„ 
Z4,., 


1 

2 


255 
0 


Zlx.«^ 

Z1..0 


Z2,.,„ 
Z2,.o 


Z3..,.,, 
Z3,.o 


Z4,.,,„ 
Z4,., 


255 


255 




Z22SS.355 




Z4,55.25, 



10 For example, memory address [(x,..,, x J , (Vn-i' Yo) J = 

[(1, 0),(255, 0)] is pixel value Zl^ stored in memory 600a. 

Once the upsampled image signal 207 has been stored 
in memories 600a-600d, it may be warped by warper 20. The 
generation of pixel value 700 in the warped image 208, shown in 
15 Fig. 1 , is described below with reference to Fig. 8. 
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First, at step 800. the address value A[x,yl of the 
source pixel value 720 in the upsampled image is generated. x 
and y are coordinates of the source pixel value 720 in the 
upsampled image and are used as an address to memories 600a- 
600d described above with reference to Tables 1 and 2. In 
response to the transform to be implemented, address generator 
620 generates an address value A[x,y] for the source pixel 
value 720 in the upscimpled image 207, shown in Fig. 7. For 
example, the input image 205 can be transformed in the spatial 
domain using a geometric transformation illustrated in 
equations (1) and (2) below. 

x[u,v] = a^ + b^»u + c^*v + d^»u2 + e,*v2 + f,*u»v ...(1) 

y[u,v] = a^ + by^u + c^^v + d^*u2 + e..»v2 + fy*u*v ...(2) 

u and y are coordinates of a pixel value in the intermediate 
warped image signal and a^, b^, c^, d^, e^, f^, a^, b^, c^, d^, e^, 
and f^ are constant coefficient values. The transform equations 
are provided to the address generator 620 via control signal 
CSl which is provided via an interface (not shown) . 
Alternatively, the transform equations can be stored in a 
memory (not shown) in address generator 620. 

Equations (1) and (2) are implemented using the 
pseudo code in Table 3 below. 



TABLE 3 

10 /*Initiali2e at the start of image warp:*/ 

x[0,0] = a, 

acceJO] = 0 

accfJO] = 0 

accejll = e^ + c^ 
20 /*Compute before the start of each line in the 

warped image : * / 

accejv] = acceJv-1] + 2e,. 

12 
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x[0,v] = x[0,v-l] + accejv] 

accf^[v] = accf^[v-l] + 
30 /*Initialize before each line of the warped 

image and after step 20: */ 
5 accdJO] = 0 

accdjl] = + d^ + accf^[v] 
40 /*Compute for each pixel in the warped image:*/ 

accdju] = accd^[u-l] + 2d^ 

x[u,v] = x[u-l,v] + accd^fu] 

10 

In Table 3, x[u,v] is the x-coordinate of address value A[x,y] 
in the upsampled image 2 07 which corresponds to a pixel value 
in the intermediate warped image signal shown in Fig. 1, This 
intermediate signal corresponds to the warped image 208 prior 
15 CO down sampling by down sampler 30 shown in Fig. 1. The y- 

coordinate y[u,v] is generated using the pseudo code in Table 3 
except x[u,v] in Table 3 is replaced with y[u,v] and the "x" 
subscripts are replaced with "y" subscripts . 

The pseudo code in Table 3 can be implemented in 
20 hardware using two adders including four accumulator registers 
for each computation to produce values x[u,v] and y[u,v]. The 
precision of the address generation is high, for example, 32 to 
4 8 bits, depending on the system requirements and accuracy 
required for the generation of the warped image 208, shown in 
25 Fig, 7. 

The address value A[x,y] has an integer part and 
and a fractional part T^ and T^ because the address value A[x,y] 
identifies the location of the source pixel 720 in the 
upsampled image 207, shown in Fig. 7, with sub-pixel precision. 

30 and are the respective integer parts of the coordinate 

values x and y and T^ and T^ are the respective fractional parts 
of the coordinate values x and y. The integer part and of 
the address value A[x,y] is used to select the two-by-two pixel 
neighborhood of the source pixel value at address value A[x,y] . 

35 The fractional part T^ and T^ of the address value A[x,y] is 
used to compute the interpolation coefficients that are 
provided to the bilinear interpolator. Interpolation 

13 
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coefficient values weight the pixel values in the two-by-two 
pixel neighborhood to produce the pixel value in the 
intermediate warped image shown in Fig- 1. 

At step 810, after the address value Atx,y] has been 
generated, it is determined whether the source pixel 720 
corresponds to one of the pixel values in the upsampled image 
signal 207. If so, the pixel value in the warped image is 
selected as the pixel value in the intermediate warped image 
signal. For example, if the address value A[x,y] identifies 
pixel value 72 0a, shown in Fig. 7, as the source pixel, then 
pixel value 720a is selected. Otherwise step 830 is performed. 

At step 83 0, the two-by-two pixel neighborhood 72 0a- 
720d of the source pixel 720 is selected. The above-referenced 
article by Gomez describes one method for selecting and 
retrieving the two-by-two pixel neighborhood from four separate 
memories . 

As shown in Figure 7, source pixel 720 is located 
within a two-by-two neighborhood of pixels 720a-72 0d which are 
used to produce the pixel value in the intermediate warped 
image signal. The four pixels 720a-720d which make up the two- 
by-two neighborhood are located in separate memories 600a-600d. 
Figure 9 is an enlarged view of a portion of the upsampled 
image 207 shown in Figure 7. Address generator 620 uses the 
upper left -most pixel as a nearest neighbor of the two-by-two 
neighborhood. The nearest neighbor is the starting position 
from which the address value for each pixel value 720a-720d in 
memories 600a-600d is determined. Figures 9(c) -9(f) illustrate 
the four different positions of source pixel 720 as it relates 
to pixel values 720a - 720d which form the two-by-two 
neighborhood. A pixel value of the upsampled image is located 
at each vertex created by the vertical and horizontal lines in 
Figures 9(c) -9(f). Figs. 9 (a) -9(d) also show in which memory 
600a-500d each pixel value 720a-720d of the two-by-two 
neighborhood is stored. 

The integer part and of the address value A[x,y] 
of the source pixel value 720 is used to determine the 
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addresses of the two-by-two neighborhood in memories 600a-600d, 
Particularly, the least significant bit of the integer part 
S^[0] and S^CO] are used to determine the addresses. When, as 
shown in Figure 9, the nearest neighbor is stored in memory 
600a, the address of each one of the pixel values 720a-720d in 
the two-by-two neighborhood is determined in accordance with 
Table 4 below. 

TABLE 4 



Memory 

Memory 600a 
Memory 600b 
Memory 600c 
Memory 600d 



X Component Addresa Y Component Addreaa 

SJll-1] SJll-1] 

SJll-1] SJll-1] 

S,[ll-1] SJll-1] 

Sjll-1] SJll-1] 



10 In Table 4, S^[ll-1] are the eleven most significant 

bits of the x component part calculated using equation (1) and 
Sy[ll-1] are the eleven most significant bits of the y component 
part calculated in accordance with equation (2). 

The addresses of pixel values 720a-720d in memories 
15 600a-600d are determined in accordance with Table 5 if the 

nearest neighbor to source pixel value 720 is stored in memory 
600b. 



20 



TABLE 5 

Memory X Component Address Y Component Address 

Memory 600a SJ11-1]+1 SJll-1] 

Memory 600b SJll-1] S^[ll-1] 

Memory 600c S,[ll-1]+1 SJll-1] 

Memory 600d S^[ll-1] SJll-1] 



The addresses of pixel values 720a-720d in memories 
600a-600d are determined in accordance with Table 6 if the 
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nearest neighbor to source pixel value 72 0 is stored in memory 
600c, 

TABIiE 6 



Memory 

Memory 600a 
Memory 600b 
Memory 600c 
Memory 600d 



X Component Addreee Y Component Addreas 

s rii-1] sjii-i]+i 



Sjll-l] 

sjii-i] 
s rii-i] 



S^[ll-l]+l 

sjii-i] • 

Sjll-l] 



The addresses of pixel values 720a-720d in memories 
600a-600d are determined in accordance with Table 7 if the 
nearest neighbor to source pixel value 720 is stored in memory 
600d. 



Memozy 

Memory 600a 
Memory 600b 
Memory 600c 
Memory 600d 



TABLE 7 

X Component Address 

SJll-11+1 
Sjll-1] 
SJ11-1]+1 
S,[ll-1] 



Y Component Address 

Sjll-11+1 
SJ11-1]+1 
S^[ll-1] 
S,[ll-1] 



After the addresses of each of the pixel values 720a- 
720d stored in memories 600a-600d are determined, the pixel 
values 720a-720d are retrieved from memories 720a-720d. 

Next, at step 840, the interpolation coefficient 
15 values II, 12, 13, and 14 are produced using the fractional 
parts T^ and T^ of address value A[x,y] . Interpolation 
coefficients II, and 12, 13, and 14 are produced using 
equations (3) -(6) below. 

II = (l-TJ • (1- T^) ... (3) 
20 12 = T,* (1- T^) . - ■ (4) 

13 = (l-TJ -Ty ... (5) 

16 
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14 = T^*T^ ... (6) 

Alternatively, interpolation coefficient values 11-14 
can be stored in a look-up table in coefficient generator 63 0. 
The fractional parts and of the address value A[x,y] are 
5 used as an address to the interpolation coefficient values 
stored in the look-up table. The above-referenced paper by 
Gomez describes one method for using the fractional parts T^ and 
Ty of the address value A[x,y] as an address for retrieving 
interpolation coefficient values from a look-up table. 

10 Returning to Fig, 8, at step 850, the source pixel 

value 720 is produced using the interpolation coefficients II- 
14 and the two-by-two neighborhood of pixels 72 0a-72 0d, shown 
in Fig. 7, retrieved from memories 600a-600d. As is shown in 
Fig. 6, pixel value 72 0a from memory 600a and interpolation 

15 coefficient value II are multiplied at multiplier 610a. Pixel 
value 720b from memory 600b and interpolation coefficient value 
12 are multiplied at multiplier 610b. Pixel value 720c from 
memory 600c and interpolation coefficient value 13 are 
multiplied at multiplier 610c. Pixel value 720d from memory 

20 600d and interpolation coefficient value 14 are multiplied at 
multiplier 610d. The output values provided from multipliers 
610a-610d are added together by sum circuitry 640 to produce 
source pixel 720, shown in Fig. 7. 

At step 860 in Fig. 8, the source pixel 720 is 

2 5 selected as a pixel value in the intermediate warped image 

signal. Steps 800 through 860 are then repeated to produce 
each pixel in the warped image 208. 

Down sampler 3 0 reduces the number of samples in the 
intermediate warped image signal by the same factor the input 

3 0 image signal is upsampled. For example, if the input image 

signal is upsampled by two, then the intermediate warped image 
is down sampled by two. Fig. 9 (a) shows respective Fourier 
transforms of the upsampled image signal 2 07, the intermediate 
warped image, and the warped image 700 illustrating the 
35 operation of the down sampler 30. Fig. 9(b) shows Fourier 
transform of a input image signal using a bilinear 

17 
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interpolation which does not include upsampling the input image 
signal. A comparison of the warped image signals shown in 
Figs. 9(a) and 9(b) illustrates that the warped image signal 
produced using upsampling prior to transformation and down 
5 sampling after transformation produces a warped signal with 
better frequency characteristics than a warped signal which 
does not use upsampling and down sampling. 

Alternatively, the down sampler 30 and the warper 20 
can be combined into a single component or step. In this case, 

10 the address value A[x,y] is adjusted by the interpolation 

factor 2** to account for the interpolation of the input image 
signal 205, For example, the standard geometric transformation 
specifies that a pixel value in the warped image signal should 
come from a pixel value in the input image signal at location 

15 A[x = XI, y = YD . This pixel value is at a non-integral 
location in the input image and, thus, is determined by 
interpolation using the neighborhood pixels. The same pixel 
value in the upsampled image signal 207 is accessed by 
multiplying the referenced address by 2 . In this case, the 

20 address value to reference in the upsampled source image would 
be A[x = 2*xl, y=2*Yl) . If the image was upsampled by a factor 
of 4, the referenced address would be multiplied by 4, Then a 
neighborhood of pixels around this address is used to determine 
the output value. The neighborhood of pixel values in the 

25 upsampled image corresponds to sample points of the original 
analog image that are closer to the location of the source 
pixel than an equal sized neighborhood of pixel values in the 
input image signal . 

Fig. 10 is another exemplary integrated circuit 
30 embodiment of the present invention directed to a warper chip 

1000. Warper chip 1000 is provided with the input image signal 
which is a multi-bit digital signal containing at least eight 
data bits and two timing bits. The two timing bits convey 
respective timing signals. One signal, HA.„, is in a logic high 
35 state when the data in a line is valid (i.e. during the active 
line interval) and in a logic low state otherwise (i.e. during 
the horizontal blanking interval). The other signal, VA,,, is 

18 
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in a logic high state when the data in a field is valid and in 
a logic low state otherwise {i.e. during the vertical blanking 
interval) . Figure 11 is a timing diagram which illustrates the 
relative timing of the input image signal and the warped image 
5 signal, and the signals HA and VA for those signals. The 
warper chip uses the timing signals HA and VA in the same 
manner as described in U.S. Patent No. 5,359,674, entitled 
PYRAMID PROCESSOR INTEGRATED CIRCUIT, issued to van der Wal , 
which is herein incorporated by reference for its teachings on 
10 processing signals having timing bits. 

' In one implementation, the timing signals HA and VA 
remain associated with the input image signal as the input 
image signal is processed by the warper chip 1000, As a 
result, when the warped image signal is produced, the vertical 

15 and horizontal blanking periods of the warped image signal do 
not need to be generated because the relative timing of the 
warped image signal is maintained by associating the timing 
signals HA and VA with the input image signal as it is 
processed. Thus, less circuitry is required. In another 

20 implementation, the upsampling and the storing process may be 
uncoupled from the warping and interpolation process, in which 
case the HA and VA timing signals for the output have to be re- 
generated. A further description of the use of the timing 
signals is omitted. 

25 The exemplary embodiment is not limited to processing 

multi-bit digital signals containing eight data bits and two 
timing bits. The exemplary embodiment can be configured to 
receive multi-bit digital signals containing, for example, ten 
data bits and two timing bits. 

30 Referring to Fig. 10, the input image signal is 

.provided to delay 1010 and upsampler 1015. Delay 1010 is the 
same as delay 200 shown in Fig. 2 and upsampler 1015 is the 
combination of vertical filter 210 and 220, shown in Fig. 2, 
and horizontal filters 230 and 240, shown in Fig. 2. Delay 

35 1010 and upsampler 1015 operate the same as these components. 

Memories 600a-600d are the same as memories 600a-600d shown in 
Fig. 6 and are a single or double bank of SRAM. Bilinear 
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interpolator is a combination of multipliers 610a-610d and sum 
circuitry 640 shown in Fig, 6. 

Address/coefficient generator 1050 operates the same 
as address generator 630 and coefficient generator 630 shown in 
5 Fig. 6 except that address/coefficient generator 10 50 may also 
receive a flow field input. The flow field input provides a 
two-dimensional vector for each output pixel. The flow field 
vector is added to the warp address generated by the 
address/coefficient generator 1050 to allow completely general 

]0 image transformations or to provide general modif icacions co 
the computed data of the address/coefficient and generator. 
Address/coefficient generator 1050 also receives control 
signals CSl from CPU interface 1040. Warp coefficients such as 
constant coefficient values a^, b^. C,, d^, e^, f^, a^., b^, C^,, d^,, 

15 e , and f^ are provided to generate address values A[x,y] . 

Address values A[x,y] are used to retrieve pixel values from 
memories 600a-600d as described above with reference to write 
address generator 615 shown in Fig. 6. Write address generator 
1020 is also coupled to CPU interface 1040 which provides data 

2C that specifies which segment of the input image is to be 

processed and stored in memories 600a-600d. Write control 
logic 1020 provides the write address shown in Tables 1 and 2 
for writing the pixel values in the upsampled image to memories 
600a-600d. Based on the timing signals HA^.^ and VA,„ and the 

25 program interface which specifies which segment of the input 
image signal should be stored, the write control logic 1020 
generates the addresses to store the pixel values in memories 
600a-600d in the same manner as described above. 

In another exemplary embodiment of the present 
30 invention, the circuitry shown in Figs 1 and 10 may be used to 
perform multiple image plane warping. In many applications it 
is desirable to warp multiple image planes such as the separate 
planes of a color image. This can be achieved by using 
multiple warper chips 1000, shown in Fig. 10, or multiple 
35 warper systems 5, shown in Fig. 1, in parallel. Alternatively, 
if the processing rate of the warper chip 1000 or warping 
system 5 is fast enough, the warping operations on part or all 
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the image planes can be interleaved. As a result, a single 
warper chip 1000 or warper system 5 can be used. 

Interleaving multi-plane images is described below 
using a RGB input image signal and Fig. 10. The input image 
signal includes, for exaanple, three image planes R, G, and B, 
where the image planes R, G, and B are interleaved for every 
pixel for each line of the input image signal as shown in Table 
8 below. 

TABIiE 8 

Line 0 R[0,C] G[0,0] B[0,0] R[1,0] G[1,0] B[1,0] 



Linen R[0,n] ,G[0,n] B[0,n] R[l,n] G[l,n] B[l,n] 



10 



IS 



By increasing line delays 1010 by a factor of three 
or using three line delays for each line delay 200, upsampler 
1015 can operate on the input image signal in an interleaved 
fashion. The upsampled image is stored in memories 600a-600d 
in the same manner as described above except that the pixel 
values are stored in an interleaved fashion. Table 9 
illustrates how the pixel values of the three planes are stored 
in memories 600a-600d. 



TABLE 9 



MEMORY 600A 



MEMORY 60 OB 



MEMORY 600C 



MEMORY 600D 



R. 



R. 



B. 



B. 



B 



0 <OI . 0{O) 



R, 
G 
B 



in. n 
m. II 
m, n 



B. 



B_ 



R, 

G, 
B, 

R, 

G, 
B. 



«, n lOJ 
n(0) 
i». n(Ot 
nvl . n<0) 
l.ntOl 



ni(0) .n (OJ 

ro<Ol . n {Ol 
m«0» ♦! ,n(01 
O) « 1, n lOl 
m(0» .1 , n (0> 



P 
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MEMORY 600A 



MEMORY 60 OB 



MEMORY 600C 



MEMORY 60 OD 



B 



■^255(01 .255 
255(01 .255 



B 



55 IO> . 255 <0» 



255I01 ,355fOI 



In Table 9 " (o) " represents the odd horizontal and 
vertical outputs of upsampler 1015. The integer part of the 
address value A[x,y] generated by address/coefficient generator 
1050 is multiplied by three to account for the change in 
address created by the addition of the other image planes in 
memories. In addition, address/coefficient generator 1050 
repeats the same operation three times: once for the calculated 
address value A[x,y], then for the address value A[x,y] 
incremented by one, and then for the address value A[x,y] 
incremented by two. The warped image signal includes color 
image data in an interleaved fashion. 

For the above example, the warper chip operates at 
three times the clock rate of the incoming pixel rate of a 
single image plane. If the RGB image planes in the input image 
signal are only available in parallel data paths, the image 
data is interleaved prior to processing. After the warper 
image signal is produced, the warped image signal is de- 
interleaved to restore the image planes to parallel data paths - 

Another example of interleaved operation of the 
warper chip 1000 is described below where the input image 
signal is an color image in Dl format, where the three image 
data planes are interleaved in a slightly different manner. 
The input image signal includes, for example, three image 
planes Y, U, and V, where the image planes Y, U, and V are 
interleaved as ...U[n,m], Y[n,m], V[n,m], Y[n+l,m], U[n+2,m], 
Y[n+2,m], V[n+2,m], Y[n+3,m]... Thus the luminance plane Y has 
twice the resolution of the two color planes U and V, and the 
color planes U and V are co-located with the even positions of 
the luminance plane Y. 

This interleaving can be maintained throughout the 
warper implementation, but the delays and address calculations 
are slightly different then when operating on the three image 
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planes as described above. The line delays 1010 prior to the 
upsampler 1015 are twice the size of a line of the luminance 
plane Y which is four times the size of one line of the color 
planes U and V. The delay switches every clock cycle between a 
5 two-delay element and a four-delay element. 

The address/coefficient generator, in turn computes a 
new address value A[x,y] twice for four successive output 
values: first for the two color planes U and V and the even 
position of the luminance plane Y, and then an address value 
10 A[x,y] is separately calculated for the odd position of the 

luminance plane Y. The address value A[x,y] is also offset to 
appropriately access the correct data planes scored in memories 
600a-600d. 

Although illustrated and described herein with 
15 reference to certain specific embodiments, the present 

invention is nevertheless not intended to be limited to the 
details shown. Rather, various modifications may be made in 
the details within the scope and range of equivalents of the 
claims and without departing from the spirit of the invention. 
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1. A image processing system comprising: 

means for applying an input image signal to the image 
processing system; 

5 means for upsampling the input image signals- 

means for warping the upsampled image signal, and 

means for down sampling the upsampled warped image 
signal to produce a warped image output signal. 

2. The image processing system according to claim 
10 1, wherein the input image signal is a digital signal sampled 

at the Nyquist rate and the upsampling means increases the 
sampling rate of the input image signal above the Nyquist rate. 

3 . The image processing system according to claim 
1, wherein the warping means includes a low pass filter and the 

15 low pass filter includes a bilinear interpolator. 

4 . The image processing system according to claim 
1, wherein the upsampling means includes a filter which 
preprocesses the upsampled image signal to counteract 
deterioration in the spatial frequency characteristics of the 

20 upsampled warped image signal caused by the warping means. 

5. The image processing system according to claim 
1, wherein: 

the input image signal includes pixel values; 

the warping means includes four memories; and 

25 the upsampling means further produces four pixel 

values for each of one the pixel values in the input image 
signal and stores each one of the four pixel values in a 
respectively different one of the four memories. 

6. A image processing system comprising: 

24 
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means for applying an input image to the image 
processing systems- 
means for upsampling the input image signal; and 

means for warping and down sampling the upsampled 
5 image signal at substantially the same time to produce a warped 
image output signal. 

7. An integrated circuit comprising: 

an input terminal to which an input image signal is 

provided; 

10 upsampler circuitry, coupled to the input terminal, 

which produces an upsampled image signal including pixel 
values; 

write address circuitry which generates a write 
address signal that specifies a location in a memory where the 
15 pixels values are to be stored; 

read address circuitry which generates a read address 
signal that specifies which ones of the pixels values are to be 
retrieved from the memory; 

warper circuitry which receives retrieved pixel 
2 0 values from the memory and produces an upsampled warped image 
signal ; and 

down sampling circuitry, coupled to the warper 
circuitry, which produces a warped image signal. 

8. The integrated circuit according to claim 7, 

25 wherein the warper circuitry includes a bilinear interpolator. 

9. A method for processing an input image signal 
comprising the steps of: 

(a) upsampling the input image signal; 

(b) warping the upsampled image signal, and 

25 
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(c) down sampling the upsampled warped image signal 
to produce a warped image output signal. 

10. The method according to claim 9, wherein step 
(a) samples the input image signal at a rate of 2" times the 

5 Nyquist rate where N is a positive integer. 

11. The method according to claim 9, wherein: 

the input image signal includes pixel values; 

step (a) produces four pixel values for each of one 
the pixel values in the input image signal, and 

10 the method further comprises the step of storing each 

one of the four pixel values in a separate memory. 
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